Skip to content

fix(agents): non-streaming fallback for Ollama agent chat#343

Merged
w7-mgfcode merged 1 commit into
devfrom
fix/ollama-streaming-chat-fallback
Jun 1, 2026
Merged

fix(agents): non-streaming fallback for Ollama agent chat#343
w7-mgfcode merged 1 commit into
devfrom
fix/ollama-streaming-chat-fallback

Conversation

@w7-mgfcode

Copy link
Copy Markdown
Owner

Root cause

Interactive Agent Chat (WebSocket /agents/streamAgentService.stream_chatagent.run_stream) fails on a local-Ollama agent: Ollama's OpenAI-compatible endpoint rejects PydanticAI's streamed request with 400 - {'message': 'invalid message content type: <nil>'} for both the primary (ollama:qwen3:8b) and fallback (ollama:llama3.1:8b), so FallbackModel raises a FallbackExceptionGroup and the UI shows "Stream error: All models from FallbackModel failed."

The non-streaming path (agent.run() in chat()) works fine on Ollama (the showcase agent_hitl_flow uses it). Cloud providers stream fine. So it's an Ollama OpenAI-compat + streaming incompatibility (a streamed-request message is serialized with content: null).

Fix

stream_chat now detects the provider from agent_default_model. For the ollama provider it uses the non-streaming agent.run() and emits the reply as a single text_delta + the existing approval_required (#336) + complete events. Cloud providers keep the true run_stream token-streaming path unchanged. The post-run logic (session persistence, deps.pending_actionawaiting_approvalapproval_required, completion) is now shared by both paths.

Tests

  • test_stream_chat_ollama_uses_nonstreaming_path — with agent_default_model=ollama:*, stream_chat calls agent.run() (asserts run_stream is not called), emits text_delta + approval_required (from a gated deps.pending_action) + complete, and flips the session to awaiting_approval.
  • test_stream_chat_cloud_keeps_streaming_path — with a cloud model, run_stream is used and run is not (regression guard).
  • Hardened 3 existing streaming tests to pin a cloud agent_default_model so they exercise the run_stream path deterministically regardless of the local .env (these were .env-coupled — they pass in CI where the default is anthropic:..., but the fix makes them robust).

Validation

  • ruff check ✅ · ruff format --check
  • mypy app/ ✅ — only pre-existing xgboost optional-dep import errors in untouched files
  • pyright app/features/agents ✅ 0 errors
  • pytest -m "not integration" ✅ 1655 passed, 12 skipped

Notes

  • Found 2026-06-01 testing the fix(agents): persist pending_action for gated tool calls #336 HITL approval card via Chat on a local-Ollama stack.
  • Compounding local limitation (out of scope): even via the working non-streaming path, small (8B) local models like qwen3:8b often don't emit the gated tool call, so the HITL card may not appear; a frontier model triggers it reliably.

Closes #342

@sourcery-ai sourcery-ai Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Sorry @w7-mgfcode, you have reached your weekly rate limit of 500000 diff characters.

Please try again later or upgrade to continue using Sourcery

@coderabbitai

coderabbitai Bot commented Jun 1, 2026

Copy link
Copy Markdown

Important

Review skipped

Auto reviews are disabled on base/target branches other than the default branch.

Please check the settings in the CodeRabbit UI or the .coderabbit.yaml file in this repository. To trigger a single review, invoke the @coderabbitai review command.

⚙️ Run configuration

Configuration used: defaults

Review profile: CHILL

Plan: Pro

Run ID: c441005d-b2ad-4cfb-b262-f06a6ced4fcb

You can disable this status message by setting the reviews.review_status to false in the CodeRabbit configuration file.

Use the checkbox below for a quick retry:

  • 🔍 Trigger review
✨ Finishing Touches
🧪 Generate unit tests (beta)
  • Create PR with unit tests
  • Commit unit tests in branch fix/ollama-streaming-chat-fallback

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

Comment @coderabbitai help to get the list of available commands and usage tips.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant